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THIS REPORT SUMMARIZES THE RECOMMENDATIONS OF A 
CONFERENCE WHICH MET TO CONSIDER WHETHER THE U.S. OFFICE OF 
EDUCATION SHOULD UNDERTAKE AN ASSESSMENT OF THE IMPACT OF ITS 
TITLE VII PROGRAM ON AMERICAN EDUCATION. IT IS RECOMMENDED 
THAT SUCH AN EFFORT INCLUDE AN OVERVIEW OF THE ENTIRE FIELD 
OF EDUCATION, WITH FOCUS ON THE FIELD OF EDUCATIONAL MEDIA. 

TO BE OF MAXIMUM VALUE, THE ASSESSMENT SHOULD BE COMPLETED AS 
QUICKLY AS POSSIBLE. A PROPOSED STUDY IS OUTLINED IN 2 PARTS. 
THE FIRST PART SUGGESTS WORK TO BE DONE — (1) AN OVERVIEW OF 
EDUCATION, WITH RECOMMENDATIONS FOR FUTURE DEVELOPMENTS, (2) 

AN ANALYSIS OF THE CURRENT STATUS OF EDUCATIONAL MEDIA, AND 
13) AN ANALYSIS OF TITLE VII RESEARCH AND DISSEMINATION 
PROJECTS. THE SECOND PART GIVES THE ESTIMATED COSTS OF THE 
STUDY. CMS) 
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I The project reported on In this publication 'was conducted pursuant to a 

| contract vith the Office of Education, U. So Department of Health, Education, 

and Welfare . The report, summarizing recommendations growing out of a confer- 
\ ence held at the Washington, D. C. office of System Development Corporation on 

1 October 7-8, 1961 for the purpose of formulating recommendations relative to a 

: possible review and assessment of research and dissemination projects conducted 

I under Title VII of the National Defense Education Act of 195®* is made to the 

Educational Media Branch of the U. S. Office of Education. 
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Guy T. Buswell, American Educational Research Association 

Nicholas A. Fattu, Indiana University 

N. L. Gage, University of Illinois 

Charles P. Hob an, University of Pennsylvania 

Arthur A. Lumsdaine, University of California, Los Angeles 

Julian Stanley, University of Wisconsin 

Robert L. Thorndike, Columbia University 

Robert L. Egbert, System Development Corporation 
David G. Ryans, System Development Corporation 
Harry F. Silberman, System Development Corporation 

J. V. Edling, Educational Media Branch, U. S. Office of Education 
C. Walter Stone, Chief, Educational Media Branch, U. S. Office of Education 
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I. SHOULD THE UNITED STATES OFFICE OF EDUCATION UNDERTAKE AN ASSESSMENT 
OF THE IMPACT OF ITS TITLE VII PROGRAM ON AMERICAN EDUCATION 



Since the inception of the National Defense Education Act, no formal over-all 
evaluation of the Title VII program has been made. The possible need for such 
an evaluation was the basis for holding this conference. 



The first question to which the Committee addressed itself was whether the 
U. S. Office of Education (OE) should undertake such an evaluation. Under 
the Title VII program, some 150 research grants have been made including 
funds for studies of educational television, programmed instruction, and the 
development and testing of materials for the different media. Some 90 con- 
tracts have been let to support dissemination activities which are intended 
to provide a research information service, a bibliographic index of media, 
normative information on use of media, leadership training, and a body of 
reports . The Committee considered whether or not a systematic evaluation 
program should be initiated to determine the actual and probable contribution 
to education of 150 research studies, of which some 25 have been completed, 
and of the many dissemination programs now under way. The direction of future 
activities of the Title VII program might well depend in part on the outcome 
of such an assessment. It was felt that results of an evaluation program 
might provide the basis for answering such questions as the following: 



1 . 



What channels should be used for research support activities 
and for dissemination activities? For example, should field 
studies be established through State Departments of Education 
or through more regional groups? What kinds of activities 
should be channeled through professional associations? 

Should research centers be established? 



2 . 



To what extent should the USOE support directed research in 
contrast to nondirected research? 



3* What types of research, in what areas, with what levels of 
• students and with what media, should be supported? 



In short, the question considered by the Committee was whether a systematic 
attempt should be made to determine where the United States is now with 
respect to educational media compared with where it might have been without 
the Title VII program and what implication this has for future development 
of Title VII activities. The Judgment of the Committee concerning a possible 
evaluation of Title VII is reflected in the following recommendations. 



1 . 



A review and assessment of the Title VII program of research 
and dissemination is both desirable and feasible. Such a 
program should be undertaken. 
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a. Projects completed and those currently in progress 
should he reviewed with attention to (l) internal 
adequacy, e.g., adequacy of design and conduct of 
program, judged importance of media developed and 
demonstrated and/or problems researched, and (2) 
social significance, or impact on the school, 
teachers, pupils, and community in terms of in- 
creased use of media, availability of improved 
techniques, and opinions of individuals related 
in various ways with the project and/or materials 
involved. (See also Appendix B, Item 1.) 

b. New research and/or dissemination projects undertaken 
under Title VII (l) should have built into their 
planning a prevision for evaluation, and (2) should 
on completion be evaluated according to a prearranged 
plan by qualified persons . Such review and evaluation 
should be directed at Internal adequacy of the research 
or dissemination project and also at the social sig- 
nificance of the undertaking. (See also Appendix B, 

Item 2.) 

2. Research and dissemination with respect to educational media 
should include projects concerned with the development and 
testing of products that may contribute to the meeting of 
needs related to the national emergency. The Educational 
Media Branch of the U . S . Office of Education should par- 
ticipate in the development of such materials .and in their 
tryout in operational settings. This will actively aid the 
program of education for defense and survival, in accordance 
with national policy. (See also Appendix B, Item 3.) 

II. PROCEDURES RECOMMENDED FOR AN ASSESSMENT OF THE TITLE VII PROGRAM 

The Committee recommended the utilization of a director for the project for 
assessing the Impact of Title VII on education. (See also Appendix B, Item 4 .) 
This person should be a leader of broad background, capable of using the tools 
of social science, including those of the historian and the sociologist. He 
should be adequately compensated and should be of the caliber of such people 
as Lee Croribach, Paul Lazarsfeld, or Chester Harris. It Is Important to note 
that technical competence is not the only criterion for this person — broad 
vision and wisdom are even more important characteristics. The project would 
gather data from a wide variety of sources and weave these data into a final 
report for submission to the U. S. Office of Education. Ideally, the report 
should be analogous to Abraham Fiexner ’ s report on medical education, the 
Smythe Report on the Manhattan Project, or the Conant study of the American 
high school. The project would undertake studies of the research projects 
which have been completed as well as of the dissemination contracts of the 
Office. Estimated time required for such an evaluation is approximately one 
year.. 
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Extensive use of penetrating interviews should he made. 1 A systematic search 
should he undertaken to find out what happens when Title VII money is expended 
in an area. Both the formulation of the questions and the interpretation of 
answers to interview questions are critical parts of the investigation; mere 
tabulation of answers to questions is not sufficient to draw out the full 
implications of the program. People who have participated in research pro- 
jects or dissemination activities can he contacted and interviewed for' an 
assessment of the impact of the program. Grantee institutions should also 
he visited and appropriate personnel interviewed. 
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Evaluation of research' projects hy consultant experts in various fields will 
also provide an important input. Data on production, distribution and use 
of materials can he obtained. Expenditures hy State Departments of Education 
and other educational organizations will provide additional data. The 
evaluation should also include data on the economics of the support program 
s 6 that a determination might he made whether to put more money into a few 
projects or less money into more projects or whether alternative support 
plans should he adopted. 

Methods for evaluating the Title VII program in contrast to other media 
support ^programs might also he used. Comparison between Title VII media 
studies and non-Title VII media studies can he made with respect to number 
and judged quality of general reports and papers, with breakdown for such 
groups as psychology journals, educational Journals, and papers and symposia 
given at professional societies. 

Although it is necessary to assess the Office of Education Programs which 
have been conducted to date, future projects, both research and dissemination, 
should have built-in evaluation mechanisms so that assessment will he conducted 
on an ongoing basis. Some projects can he tested immediately; for example, a 
film should he accompanied hy a test on its content. Before the final research 
report comes out on a project, feedback to the grantee from qualified critics 
should he supplied. Such a review would help the researcher Improve his report 
and would also supply the Office of Education with an immediate evaluation of 
the study. At the end of the dissemination program, teachers and administrators 
should he asked what they think of it. Such feedback should he cumulative and 
have a control function on subsequent proposals. Procedures for assessing the 
OE program' should he aimed at its improvement over the next few years . 



The remainder of this section consists of the Committee* s suggestions for 
implementing the assessment. In making these suggestions, the Committee 
was not in any sense attempting to dictate procedures to the project 
director. Rather, the Committee members were making their ideas available 
to he used as appropriate. 
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XU, POSSIBLE OUTCOMES AND EXTRAPOLATIONS TO FUTURE PROGRAMS 

In attempting to determine whether a study should he conducted, a prime con- 
sideration is the utility, broadly defined, of the possible outcome. While 
the Committee did not formally enumerate anticipated results of the proposed 
study, numerous suggestions of probable findings were made during the course 
of the meetings • These predictions , which are summarized below, indicate 
that the Committee was. giving serious attention to this facet of the problem. 
In reviewing the following presentation, the reader should keep in mind that 
it is not intended as a set of statements of fact. These are predictions of 
outcomes and, from that frame of reference, may be of value in determining 
whether the evaluative study should be made* 

It is predicted that an evaluation study will find that: 

1 . Title VII, including research on and dissemination of 
information about educational media, has served to improve 
instruction in public elementary and secondary schools and 
institutions of higher education and should be continued. 

2. Research and dissemination projects which deal with short- 
term or national emergency problems should be supported. 

This support should not operate to the exclusion of more 
long-range objectives. 

3 . There should continue to be representation of each of the 

various educational media among the projects supported. j 



- 6 - 




Preference should be given to projects which promise to move 
in an orderly sequence through the following stages: 

a. hypothesis discovery 

b . hypothesis testing 

c. field suitability testing 

d. administrative logistics testing 

e. program and media production 

f. all-out demonstration 

Preference should also be shown to studies initiated anywhere 
in this sequence when it is demonstrated that earlier steps 
have been reasonably well accompli shed. 



1 See "Recommendations for Research and Development Efforts in Programmed 
Instructiori' (13 October 3-961 ) by Joint Committee on Auto-Instructional 
Programs and Devices for AERA-APA-DAVI . These recommendations are also 
pertinent to points 5 and- 6. 
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5. Support should he given to investigators who are careful, 
insightful workers with adequate facilities for their studies. 

6. Projects should explore new ways in which to use media rather 
than simply compare media with conventional methods. At- 
tention should he given to working with young children and 
to teaching them subject matter other than the conventional. 

7 • Preference should he given to creative studies with high risk 
hut high payoff rather than low risk, low payoff. 

8. More support should he devoted to the training of research 
workers . 

9. The preferred variables to he studied are psychological 
(e.g., cognitive or medlational) rather than grossly 
physical, sociological, or physiological (e.g., size of 
screen, size of class, sex of student). In any event. 
Investigators should engage in explicit "psychologizing 11 
about their work, .both before and after their results have 
been obtained. 

10 . Studies in which time is a dependent variable should be 
fostered. 

11 . More effort should be expended toward disseminating research 
findings to educational administrators and classroom teachers. 



1 While the preceding predictions are not all-inclusive, they should provide 
3 some concept of the range of outcomes possible from a study such as the one 
I recommended. 
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APPENDIX A 

CONSIDERATIONS FOR A NATIONAL EMERGENCY EDUCATIONAL PROGRAM 



.Recommendations and suggestions relative to possible gaps in the program of 
currently sponsored research and dissemination under Title VII 'were discussed 
in two major contexts, or in light of two time perspectives: (l) a perspective 

which envisaged a continuing normal development of educational activities over 
the period of the next two decades, and (2) a perspective which recognized the 
cold war climate of American political and economic thinking circa 1961 and the 
possibility of a national emergency in the immediate future. Recognizing the' 
second perspective and the relative state of ignorance of the population with 
respect to means of surviving and carrying on in the event of thermonuclear 
attack, the Committee suggested the active participation of the Educational 
Media Branch in providing instructional materials directed at civil defense, 
before, during, and following possible attack. It was suggested that the 
Educational Media Branch consider a two-pronged approach to preparation for 
possible national emergency: 

1. Encouragement of research and dissemination projects geared 
to develop educational materials likely to be most useful in 
this short-time perspective and to test the effectiveness of 
such materials. 

2. Direct contracting for the production of such materials as 
film programs which would be widely distributed to schools 
and communities for educating the population regarding 
(a) protection, (b) first aid, (c) survival, and (d) 
continued participation in national affairs in the event 
of thermonuclear attack. 

It was suggested that it might be advisable to consider such programs, i.e., 
defense and survival, from the standpoints of different time periods (e.g., 
imminent attack, attack within six months, attack within one year, attack 
within five years) and to develop a matrix of goals and tasks relative to 
educational media adapted to the different time periods. With the shorter 
time perspective, it might be expected that attention would focus on in- 
structions more than instructing, and in development, production and imple- 
mentation more than research and study. With longer time perspective, 
attention might well be devoted to an increasing degree to study of the 
effectiveness of different kinds of materials and media, to the education 
of the public in protection, survival, and the like. 
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APPENDIX B 

ADDITIONAL SUGGESTIONS 



In accordance with prior plans, the draft of the Conference Report was sent to 
the participants for their comments. Most of the comments made were directly 
useful in preparation of the final report. However, there were a few major 
suggestions which, if included in the hody of the report, appear possibly to 
he at variance with the majority opinion. These suggestions constitute the 
content of Appendix B . 

1. In discussing Item l.a., Page 4, one participant suggested 
that it would he possible to satisfy both criteria and still 
not develop a body of dependable knowledge. He proposed in- 
clusion of a third criterion for judging Title VII activities. 

This criterion would be "potential contribution to dependable 
knowledge." 

One of the participants felt that recommendation (2) of Item l.b.. 
Page 4, poses a threat to "initiative and responsibility in re- 
sponsibility in research." This participant favored evaluation 
but would not prescribe "expert" evaluation as the method nor 
would he prescribe any single method for all projects. 

3* One of the participants recommended an even stronger statement 
of Appendix A and of other suggestions related to "our national 
survival." In this connection, he suggested that consideration 
be given not only to our own civil defense but also to "projects 
likely to have payoff in meeting the pressing educational re- 
quirements of the uncommitted and/or underdeveloped nations of 
the world." This participant also recommended the "advisability 
of a systematic planning activity . . . for the survey and analysis 
of educational requirements in the light of the world situation, 
with reference to the kind of time perspectives outlined in 
Appendix A ... and with the intent of developing policies, 
priorities and plans appropriate to the best possible estimate 
of contingencies in relation not only to all-out thermonuclear 
war but also to the prospects of a long-protracted total world 
struggle on all fronts, with national survival as the stake." 

4. It was emphasized by one participant that the effort recommended 
by the conference requires significant organizational support. 

The implication of this, he felt, is that Title VII should 
contract with an organization which could provide or hire such 
an individual and could furnish him such support as he requests. 
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APPENDIX C 



SDC EVALUATION 



Analysis of the problems confronted in considering the desirability of a 
thorough review and assessment of projects initiated under Title VII of 
the National Defense Education Act convinces the Educational Research and 
Development staff of System Development Corporation that any attempt at such 
an assessment should include an overview of the entire field of education, 
with focus on the field of educational media. The more complete this overview, 
the more valuable wi**! be the evaluation of Title VII projects. 



Balanced against the need for an overview of education as a foundation for 
an appraisal of Title VII projects is the time factor. To be of maximum 
value to those recommending future Office of Education work with educational 
media, the proposed evaluation of Title VII projects should be completed as 
expeditiously as possible. 

In an endeavor to satisfy both requirements — (l) an early report (2) based 
on an extensive view of education — the following study, with estimated 
cost, is recommended by SDC. Part I consists of the recommendation of work 
to be done. Part II gives an estimate of cost. 



I 



I 




I. RECOMMENDED WORK 

A. Overview of Education 



Of the several recommendations of Appendix C, this "overview 
of education" is the most difficult, the most time-consuming 
and the most expensive. It also is indispensable for at- 
tainment of the goals of an evaluation. Although an appraisal 
of the present status of educational media and its relationship 
to Title VII work undoubtedly would be of value, the real key 
to the SDC recommendations is an overview of education in 
general. Only as the totality of education is examined can 
the specific roles of educational media and Title VII projects 
in particular be viewed with precision. Furthermore, to give 
real meaning to an examination of education, it is necessary 
that this scrutiny involve the future as well as the past and 
the present. 

To accomplish this goal, it is recommended that a number of 
exceptionally bright , capable, creative education-oriented 
individuals be brought together for an extended period of 
time, be asked to examine the current educational scene. 
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and be requested to give a description of what education 
should be ten years from now. (Such scholars could 
probably obtain two-year leaves from their institutions 
for participation in such a study.) These carefully 
chosen persons should have access to facilities and 
personnel for conducting research which may be critical 
to their deliberations. This aspect of the study, which 
would run concurrently with the analyses noted in recom- 
mendations B and C below, would require perhaps two and 
one-half years to complete. 

This aspect of the appraisal would provide: 

1. A summary statement of the current status of 
American education. (This could be based in 
part on work already completed by other groups . ) 

2. A recommendation for future direction of American 
education. 

3* A statement of the idealized role of educational 
media. 

B. Educational Media 

It is recommended that ah analysis be made of the current 
status of educational media. This would begin at an early 
date and might require approximately 18 months to complete. 

A report from this analysis would include: 

1. A description of available media. 

2. A review of recent research, perhaps since 1950. 

3. A description of the status of dissemination of 
information concerning educational media. 

4. An account of the present utilization of educational 
media. 

C. Analysis of Title VII Research and Dissemination Projects 

It is recommended that a detailed, analytical report be 
prepared of both the research and dissemination aspects 
of Title VII. This work would also begin at a very early 
date, and would run concurrently with the analysis of 
educational media, thus enabling the two projects to provide 
feedback to each other. A comprehensive report of this 
examination would include summarizations and recommendations 
based upon: 
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1. Intensive interviewing of a representative sample 
of individuals receiving research grants and dis- 
semination contracts. 

2. Gathering of questionnaire information from all 
grant and contract recipients. 

3. Evaluation of research projects hy competent 
consultants. 

4. Assessing effect of the dissemination program 
through analysis of such sources as distribution 
lists, through questioning samples of those re- 
ceiving media information under the Title VII 
dissemination program, etc.. 

It is envisaged that the three aspects of the proposed 
evaluation would begin at approximately the same time and 
would proceed concurrently. The comprehensive review of 
education would, however, require longer to complete. In 
the final report of the comprehensive review a rapprochement 
would be attempted between the thinking in this area and that 
involved in the more limited aspects of the evaluation, i.e., 
the analysis of educational media and the analysis of Title 
VII research and dissemination projects. 



II. ESTIMATED COST 

Although any financial figures suggested are of necessity rough estimates, 
it is believed that, within the time limitations imposed, reasonable progress 
could be made by the agency engaged to accomplish the defined goals with a 
budget approximating that which follows. 



Administration 
Title VII Project Analysis 
Analysis of Educational Media 
Overview of Education 



1,515,000 



115,000 

235,000 

160,000 



Total 



2,025,000 




